Skip to content

[FLINK-38888][transform] YAML Pipeline supports item subscription of array, map and row#4241

Merged
yuxiqian merged 3 commits into
apache:masterfrom
Hisoka-X:add-array-and-map-access
Feb 4, 2026
Merged

[FLINK-38888][transform] YAML Pipeline supports item subscription of array, map and row#4241
yuxiqian merged 3 commits into
apache:masterfrom
Hisoka-X:add-array-and-map-access

Conversation

@Hisoka-X
Copy link
Copy Markdown
Member

@Hisoka-X Hisoka-X commented Jan 29, 2026

This pull request adds support for accessing elements within ARRAY and MAP types using bracket notation in SQL expressions, following the SQL standard (1-based indexing for arrays). It introduces a new utility method for item access, updates the expression parser to handle this syntax.

Due to limitations in the Calcite syntax, row access will be implemented separately later.

@github-actions github-actions Bot added docs Improvements or additions to documentation runtime labels Jan 29, 2026
@Hisoka-X
Copy link
Copy Markdown
Member Author

Hisoka-X commented Jan 29, 2026

cc @yuxiqian

@yuxiqian
Copy link
Copy Markdown
Member

yuxiqian commented Jan 29, 2026

Hi Hisoka, thanks for your contribution! Considering currently transform module couldn't handle nested types well, I would suggest merging #4236 first and work on this ticket later. WDYT?

Plus, there's some disabled test specs in flink-cdc-composer/src/test/resources/specs/nested.yaml. It would be great if this PR could make these cases pass.

@Hisoka-X
Copy link
Copy Markdown
Member Author

Hisoka-X commented Jan 29, 2026

Hi Hisoka, thanks for your contribution! Considering currently transform module couldn't handle nested types well, I would suggest merging #4236 first and work on this ticket later. WDYT?

OK

Plus, there's some disabled test specs in flink-cdc-composer/src/test/resources/specs/nested.yaml. It would be great if this PR could make these cases pass.

Let me enable it to verify.

@Hisoka-X Hisoka-X marked this pull request as draft January 29, 2026 03:48
@yuxiqian yuxiqian mentioned this pull request Jan 30, 2026
Merged
@Hisoka-X Hisoka-X force-pushed the add-array-and-map-access branch from 0c277b5 to 0086aa3 Compare February 2, 2026 07:21
@Hisoka-X Hisoka-X marked this pull request as ready for review February 2, 2026 07:21
@Hisoka-X
Copy link
Copy Markdown
Member Author

Hisoka-X commented Feb 2, 2026

Hi @yuxiqian , please help to review this pr. Thanks.

@Hisoka-X Hisoka-X changed the title [FLINK-38888][transform] YAML Pipeline supports item subscription of array and map [FLINK-38888][transform] YAML Pipeline supports item subscription of array, map and row Feb 2, 2026
yuxiqian
yuxiqian reviewed Feb 2, 2026
View reviewed changes
Copy link
Copy Markdown
Member

@yuxiqian yuxiqian left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for your update! Just left some comments.

Comment thread ...k-cdc-runtime/src/main/java/org/apache/flink/cdc/runtime/functions/impl/StructFunctions.java Outdated
Comment thread ...k-cdc-runtime/src/main/java/org/apache/flink/cdc/runtime/functions/impl/StructFunctions.java Outdated
Comment thread flink-cdc-runtime/src/main/java/org/apache/flink/cdc/runtime/parser/TransformParser.java Outdated
Comment thread flink-cdc-runtime/src/main/java/org/apache/flink/cdc/runtime/parser/JaninoCompiler.java Outdated
Comment thread flink-cdc-runtime/src/main/java/org/apache/flink/cdc/runtime/parser/JaninoCompiler.java Outdated
Comment thread ...me/src/main/java/org/apache/flink/cdc/runtime/parser/metadata/TransformSqlOperatorTable.java Outdated
Comment thread ...me/src/main/java/org/apache/flink/cdc/runtime/parser/metadata/TransformSqlOperatorTable.java Outdated
Comment thread ...me/src/main/java/org/apache/flink/cdc/runtime/parser/metadata/TransformSqlOperatorTable.java
yuxiqian
yuxiqian approved these changes Feb 3, 2026
View reviewed changes
Copy link
Copy Markdown
Member

@yuxiqian yuxiqian left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM.

@yuxiqian yuxiqian requested a review from lvyanquan February 3, 2026 01:19
@yuxiqian yuxiqian merged commit 09ad789 into apache:master Feb 4, 2026
43 of 46 checks passed
@Hisoka-X Hisoka-X deleted the add-array-and-map-access branch February 4, 2026 02:41
ThorneANN pushed a commit to ThorneANN/flink-cdc that referenced this pull request Mar 19, 2026
@Hisoka-X @ThorneANN
[FLINK-38888][transform] YAML Pipeline supports item subscription of …
5b3c060 
…ARRAY, MAP, and ROW (apache#4241)
Mrart pushed a commit to Mrart/flink-cdc that referenced this pull request Mar 26, 2026
@Hisoka-X @Mrart
[FLINK-38888][transform] YAML Pipeline supports item subscription of …
66ed84f 
…ARRAY, MAP, and ROW (apache#4241)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@yuxiqian yuxiqian yuxiqian approved these changes

@lvyanquan lvyanquan Awaiting requested review from lvyanquan

Assignees

No one assigned

Labels

approved composer docs Improvements or additions to documentation reviewed runtime

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@Hisoka-X @yuxiqian